Field elimination apparatus for a video compression/decompression system

ABSTRACT

Video signal compression apparatus includes a memory for storing two fields of image data and providing fields of image data separated by one frame intervals. Image data from corresponding fields of successive frames are subtracted to generate field differences. The field differences are accumulated over respective field intervals, and the sum is compared against a predetermined value. If a sum of differences over a field is less than the predetermined value, the most recent field is considered to be redundant and may be excised. After respective fields are excised, frames of video signal are composed from the remaining fields. The field types (e.g., odd or even) of the fields in the memory are determined, and the data is accessed from the respective fields in memory such that data from odd and even fields occupy odd and even lines in the composed frames respectively. Flags are associated with composed frames to indicate which fields contain data associated with excised fields, and which field of the composed frames should be displayed first.

This invention relates to apparatus for processing video signal prior tosignal compression and post decompression. More particularly it relatesto apparatus for eliminating redundant fields of video signal (at anencoder) and restoring (at a decoder) fields of video signal that havebeen excised by an encoder.

BACKGROUND OF THE INVENTION

Currently much effort is being expended in developing systems forcompressing video signals for both television signal transmission, andmulti-media purposes, i.e. computer usage. This effort is typified bythe ongoing development of a compressed video standard, which iscurrently identified as MPEG. (MPEG stands for "Moving Pictures ExpertGroup", which is a committee of the ISO, the International OrganisationFor Standardisation.) MPEG is a compressed video signal protocoldescribing a hierarchically layered motion compensated, Discrete CosineTransformed transformed, Quantized, statistically encoded etc. signalformat, a draft of which is identified as ISO documentISO-IEC/JTC1/SC29/WG11 MPEG 92/160. The MPEG protocol requires thatvideo signal be encoded (compressed) on a frame basis in groups offrames, GOP's. The compressed signal for a GOP includes a GOP headerfollowed by a frame header, followed by a slice header (a slice being aportion of a frame comprising, for example, 16 horizontal rows),followed by a macroblock header, (a macroblock being for example a 16×16matrix of pixels) followed by blocks of pixel data, followed by the nextframe header and so on.

Nominally a compression apparatus arranged to provide compressed dataaccording to the MPEG format, receives data either on a field or framebasis, groups this data into GOP's, and performs the requisitecompression processes to generate a MPEG signal. Note however, if thesource material is video signal originally produced on film andconverted to video via a Telecine, that is by 3:2 pulldown, one fieldout of five is redundant. Removal of the redundant fields from suchmaterial immediately provides a twenty percent increase in compressionefficiency. Further, even if the video signal was produced by a videocamera, much of this video signal may represent still images, in whichcase frames of data may be redundant. Elimination (at least in part) ofsome of the still image redundancy will also effect an increase incompression efficiency.

In order to realize such enhancements to compression efficiency, it isnecessary to identify image field redundancies and thereafter excisethese fields prior to application to the compression apparatus. Further,after fields have been excised, a system of identifying excised fieldsmust be implemented to inform the reciprocal decoding apparatus toregenerate the excised fields. The present invention is directed toapparatus for detecting redundancy in video image fields, excising onesof the redundant video image fields, and encoding the remaining data ina fashion to enable a decoding apparatus to restore the excised fields.

SUMMARY OF THE INVENTION

The present invention includes a memory for storing two fields of imagedata and providing fields of image data separated by one frameintervals. Image data from corresponding fields of successive frames aresubtracted to generate field differences. The field differences areaccumulated over respective field intervals, and the sum is comparedagainst a predetermined value. If a sum of differences over a field isless than the predetermined value, the most recent field is consideredto be redundant and may be excised form the video data stream.

After respective fields are excised from the video signal stream, framesof video signal are composed from the remaining fields. The field types(e.g., odd or even) of the fields in the memory are determined, and thedata is accessed from the respective fields in memory such that datafrom odd and even fields occupy odd and even lines in the composedframes respectively. Flags are associated with composed frames thatcontain data corresponding to excised redundant data. Further flags aregenerated to indicate which field in a frame including such redundantdata contains the redundant data. The flags are incorporated into thecompressed video data with the associated frames after the frames ofvideo data are compressed.

At the receiver portion of a video signal compression system, thecompressed signal is examined and any field redundancy flags areremoved. The compressed data is then decoded and decompressed videosignal is loaded into a display memory. Subsequently the display memoryis read to provide an image in raster format. The memory controlapparatus is made responsive to respective redundancy flags to repeatthe display of associated fields of image data corresponding to excisedfields.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial representation and waveform diagram of a sequenceof video signal input flames and reconstituted flames, whichrepresentation is useful in describing the invention.

FIG. 2 is a pictorial representation and flag signal diagram of asequence of video signal output flames provided by a decoder apparatus,and a pictorial representation of resequenced fields of video signal forproviding an interlaced video signal, which representations are usefulin describing the invention.

FIG. 3 is a block diagram of apparatus for excising redundant fields ofvideo signal and generating flag signals.

FIG. 4 is a block diagram of a resequencer which may be utilized in theFIG. 3 apparatus.

FIG. 5 is a flow chart detailing the functional operation of theresequencer of FIGS. 3 and 4.

FIG. 6 is a block diagram of decoder apparatus for generating substitutevideo fields for excised redundant video fields.

DETAILED DESCRIPTION

Refer to FIG. 1 and the row of boxes designated input video. Each columnof circles represents a field of video signal, with each circle in afield representing a horizontal line. In that portion of the row ofboxes designated "video", the respective boxes encompassing two suchfields represent frames of interlaced video signal generated by, forexample, a video camera. The portion of the row of boxes designated"film", constitutes video signal developed by, for example, a telecinewherein one of every four occurring fields is reproduced in the signalstream. The repeated fields are incorporated in the boxes encompassingthree fields. For example, in the box including the fields L, M and N,field N is a repeat of field L.

The row of boxes designated output frames represents resequenced framesof video signal after redundant fields have been excised. In this figureit is assumed that the "video" frames contain moving images, andtherefor only "film" frames will have redundant fields of information,whether or not the "film" sequences contain moving images. In theFIGURE, the resequenced fields are arranged in non-interlaced frames forapplication to, for example, an MPEG encoder. It can be seen that forevery five fields of input signal designated "film" frames, the outputframes include only four fields, effecting a 20% data reduction.

The input video signal could just as well represent still "video", inwhich case successive even fields will be identical except for noisecontributions, and successive odd fields will be identical except fornoise contributions. In the no-motion or still video signal instance,the redundant fields may occur randomly or bunched together. If theredundant fields occur randomly, the present system will excise theredundant fields as they occur. If, on the other hand, still fieldsoccur in relatively large bursts, the preferred embodiment of theinvention will only excise m of every n redundant fields, where m and nare integers with n greater than m. This constraint is imposed topreclude the system rate buffer from underflowing. Note, representativevalues for n and m are 3 and 1 or 5 and 1, etc.

A signal DT indicates the ones of the resequenced frames that include afield corresponding to a field which has been excised. A representationof the signal DT will be included in the transmitted encoded signal toinform the receiver that one of the fields in the associated frameshould be displayed twice. A second signal DF indicates which field inevery frame is to be displayed first by an interlace display system. Arepresentation of the signal DF is also included in the transmittedencoded signal. It will be noted from the input and output videofields/frames that even fields nominally occur in time after odd fields.That is an interlace frame includes an odd field interlaced with asubsequently scanned even field. However when video data is excised on afield basis, reconstructed output frames may include an odd field with aprior occurring even field. Hence it is necessary to keep track of whichfield in a reconstructed frame occurred first. In other words, when afield is repeated at the receiver as a substitute for an excised field,it is necessary to know which field of the subsequent frame should beinterlaced with the repeated field.

In FIG. 2 the row of boxes designated input frames represents the outputframes of FIG. 1 after having been decoded in an appropriate receiver.Associated with the respective frames are the corresponding signal DTand DF represented as two-bit binary words. The leftmost and rightmostbits represent the signals DT and DF respectively. A one in the left bitposition indicates that the associated frame includes a redundant field.A one and a zero in the right bit position indicates that the even andodd fields respectively, of the associated frame is to be displayedfirst.

The row of boxes designated Output Fields represents the sequence thatthe fields comprising the input frames should be displayed.

FIG. 3 illustrates apparatus for detecting redundant fields, excisingredundant fields and reconstructing frames of video signal from theremaining fields. Video signal is applied from a source 10 to a firstbuffer memory (B_(n)) 12 and a subtracter 16. An output from the buffermemory 12 is coupled to a second buffer memory (B_(n-1)) 14. The outputof the second buffer memory is coupled to the second input of thesubtracter 16. The first and second buffer memories each delay signal byone field period, hence the two video signals applied to the subtracter16 correspond to like spatial positions of like field types separated byone frame interval. If two fields of video signal separated by a frameare identical (excluding noise) the differences provided by thesubtracter 16 over respective field periods will be zero (assuming thevideo signal represents luminance only). The differences developed bythe subtracter 16 are applied to a coring circuit 17, which eliminatessignal differences less than a predetermined value in order to minimizethe effects of signal noise in the video signal differences. The coreddifference values are applied to an accumulator 18 wherein the absolutevalues of the differences are accumulated (summed) over respective fieldintervals. The summed values are applied to a threshold detector 20wherein they are compared to a threshold value, and if the sum ofdifferences for a respective field is less than the threshold value, thefield stored in buffer memory 12 is considered redundant with thelastmost previous field of the same type, that is the last field readout of buffer memory 14. If the sum is greater than the threshold value,the field currently stored in buffer memory 12 in considerednon-redundant.

Video data from the buffer memories 12 and 14 are applied to aresequencer 21, and video data from the source 10 are applied to theodd/even field type detector 11. The resequencer 21 responsive to fieldtype data from the detector 11, and data from the threshold detector 20,excises and reformats the fields of data from the source 10. Theresequenced video signal data is applied to a compression apparatus 23which may include a motion compensated predictive encoder associatedwith a Discrete Cosine Transform encoder, and statistical and run lengthencoders. Compressed video signal provided by the compression apparatus23 is applied to a data formatter 24. The data formatter formats thecompressed data with ancillary data for synchronization and/or errordetection/correction.

If the compressed video signal is to be transmitted over a medium otherthan copper wire, for example, it will be necessary to provide thecompressed video signal with further noise protection. Hence thecompressed data from the data formatter 24 is applied to a transportprocessor 26 which adds signal redundancy. This redundancy is specificto certain types of data critical to signal decoding. The transportprocessor 26 forms packets of data having a payload of a predeterminednumber of compressed video bits, and flexible headers including datawhich identifies the spatial location within respective images fromwhich the payload data was derived. For further information of transportprocessors of this type see U.S. Pat. No. 5,168,356 entitled "Apparatusfor Segmenting Encoded Video Signal For Transmission".

The formatter 24 and the compression apparatus 23 operate under thecontrol of a compression controller 22. The controller 22 receives videodata and display data (DF) from the resequencer 21, and data DT from thethreshold detector 20. The controller 22 will nominally operate as astate machine to condition the compressor to provide data in apredetermined sequence, and to condition the formatter 24 to layer theoutput data in a predetermined hierarchy, e.g. according to the MPEGsignal protocol. If in fact the controller and formatter are programmedto provide MPEG formatted data, the aforementioned flag signals DT/DFwill not be introduced in the data stream by the formatter 24. Thesignals DT/DF will be provided to the transport processor 26 with theassociated frames of compressed video data. The signals DT/DF maythereafter be incorporated in respective transport packet headersassociated with transport packets conveying payloads which include FrameHeader information.

Alternatively, if the encoder 25 is not an MPEG encoder, but perhaps amodified MPEG+ encoder, provision may be made to incorporate the signalsDT/DF within the frame headers of the modified MPEG compressed data. Inthis instance the controller 22 and formatter 24 will be prearranged toincorporate the signals DT/DF directly into the compressed video datastream. It will be appreciated by those skilled in video signalcompression techniques, and armed with the foregoing disclosure, thatvaried other methods may be implemented for conveying the respectiveflag signals associated with fields/frames conveying redundant data.

FIG. 4 illustrates exemplary apparatus for the resequencer 21 of FIG. 3.The resequencer 21 of FIG. 4 forms frames from fields of video data.That is, it interleaves horizontal lines of even fields with horizontallines of odd fields to form a frame from two fields. The interleaving isperformed by the register pairs 30, 31 and 32, 33. Buffer memories 12and 14 will always be conditioned to contain the two fields to becombined into respective frames. The field memories 12 and 14 will beread out concurrently, a line at a time. Respective lines read out ofmemory 12 are written to a serial-input-parallel-output register 30,having parallel output connections coupled to parallel input connectionsof a parallel-input-serial-output register 31. After a respective lineis written to register 30, it is loaded into register 31. Thereafter theline is serially read out of register 31. Similarly respective linesread from memory 14 are written to register 32, transferred to register34, and then serially read out of register 34. The output signalsprovided by registers 31 and 33 are coupled to respective inputconnections of a multiplexer 34. The multiplexer 34 is conditioned bythe resequence controller 35 to couple even field lines in even fieldline positions and odd field lines in odd field line positions withineach composed frame.

In this exemplary embodiment, data is read from the field memories 12and 14 at a predetermined sample rate a line at a time. Thereafter theregisters 31 and 32 are clocked mutually exclusively, under control ofthe controller 35, at twice this sample rate such that the signal outputfrom the multiplexer consists of a line of video signal from register 31(33) followed by a line of video signal from register 33 (31). Asrespective fields are applied to the buffer memories, the field detector11 determines whether the fields are odd or even and conveys such fieldtype information to the controller 35. Controller 35 therefore knowswhich of buffer memories 12 and 14 contain the odd and the even field,and thus may appropriately switch the multiplexer 34. The controller 35is also supplied with data from the threshold detector and is arrangedto generate the signal DT/DF.

FIG. 5 is a flow chart of the operation of the FIG. 4 resequencer, whichoperates as follows. At system startup two consecutive fields are loaded[100] into the buffer memories 12 and 14 and an index n is set to 2. Theresequencer then fetches [101] the field type (even or odd) of the fieldloaded in memory 12. The field type is examined [102], and if the nextfield should be an even (odd) field the system branches to path 103,104, 105 (107, 108, 109).

If even, the index n is incremented by one, and loading of the nextfield into memory 12 is initiated [103]. If even then the fieldcurrently residing in memories 12 (B_(n)) and 14 (B_(n-1)) are odd andeven respectively. The current frame is built [104] from the fieldscurrently in memories B_(n) and B_(n-1), with lines of video signal frommemory B_(n) as odd lines and lines of video signal from memory B_(n-1)as even lines. The odd and even fields in memories 12 and 14 are inreverse time order and thus the signal display first (DF) is set to alogic one value [105].

If odd, the index n is incremented by one, and loading of the next fieldinto memory 12 is initiated [107]. If odd then the field currentlyresiding in memories 12 (B_(n)) and 14 (B_(n-1)) are even and oddrespectively. The current frame is built [108] from the fields currentlyin memories B_(n) and B_(n-1), with lines of video signal from memoryB_(n) as even lines and lines of video signal from memory B_(n-1) as oddlines. The even and odd fields in memories 12 and 14 are in normal orderand thus the signal display first (DF) is set to a logic zero value[109].

After establishing the state of the signal DF, The signal DT from thethreshold detector is loaded [110] and examined [111] for the currentfield. If the signal DT is low indicating no field redundancy, the nextfield is loaded [115] into memory so the memories 12 and 14 contain anew frame of data. The system then returns to step [101].

If the signal DT is high indicating field redundancy, a variable last₋₋drop (LD) is examined [112]. The variable LD keeps track of the mostrecently dropped field. The variable LD is subtracted from the index n,and the difference is compared to a further predetermined value drop₋₋frequency FD. The value FD determines the number of fields that arepermitted to be dropped per sequence of fields. The lowest valid valueof FD is 3 permitting one in three fields to be dropped. An FD value of5 will permit no more than one in five fields to be dropped. If thedifference (n-LD) is less than or equal to FD, DT is set to zero [113]and the system branches to step 115. In this instance the systemprecludes the dropping of a field even though it is redundant.Alternatively if the difference is greater than FD, then LD is set to n[114] and the next two fields are loaded [116] into the memories 12 and14. This has the effect of dropping the field currently residing in thememory 14. The system then branches back to step [101].

FIG. 6 illustrates an exemplary embodiment of a receiver prepared toutilize the flag data DT/DF to reconstruct video data in which redundantfields have been excised. Compressed video data from a transmissionpath, for example a tuner is applied to a transport processor 60. Thetransport processor receives the transport packets of compressed videoand synchronizing information etc., separates the compressed videosignal from the transport packet headers, and applies the compressedvideo signal to an appropriate decoder/decompresser. If the redundantfield flag data DT/DF is contained in the packet transport headers, thisflag data is separated and applied to the system controller 64, with anyother ancillary data needed by the system controller 64. If the flagdata DT/DF is contained in the compressed video signal per se', it isseparated by the decoder/decompresser 61, and applied to the systemcontroller 64. The system controller ultimately directs the signal DT/DFto a memory controller 66.

Compressed video signal is decompressed by the decoder 61 and loadedinto a display memory 62. Nominally the display memory 62 will contain aframe of decompressed data. The decompressed data is coupled from thedisplay memory to an output display device (or to a recording deviceetc) via a multiplexer 63. When no field redundancy is indicated, datais coupled directly from the memory 62 to the output device.

The output of the display memory is also coupled to an additional fieldmemory 65, and the output of the field memory 65 is coupled to a secondinput of the multiplexer 63. The display memory 62, field memory 65 andmultiplexer 63 are controlled by a memory controller 66. The memorycontroller 66 is conditioned by the system controller 64, includingsignals DT/DF, to provide video data from the respective memories fordisplay. If no field redundancy is indicated by the signals DT/DF,decompressed odd and even fields are respectively loaded into thedisplay memory in odd and even field positions responsive to the signalDF. The odd and even fields are then consecutively read from the memory62 in interlace fashion.

If a field associated with redundant information is indicated by thesignal DT/DF, when such field is read out to the display device it isconcurrently captured in the field memory 65. Thereafter the field ofvideo signal in the field memory 65 is coupled to the display device bythe multiplexer 63 in the appropriate field position. Depending on theparticular decompression system, it may not be necessary to include anextra field memory 65 and multiplexer 63, for field repeats, and therepeating of fields of video signal may be performed directly from thedisplay memory 62. The latter arrangement reduces the amount ofnecessary hardware, but increases timing constraints on both the displaymemory and the decompresser 61.

What is claimed is:
 1. Apparatus for receiving a compressed video signalwherein ones of substantially redundant fields of video signal have beenexcised and frames of video signal were reconstructed from the remainingfields after said substantially redundant fields were excised, saidcompressed video signal including information DT/DF indicative of fieldsof compressed video signal having been excised and the order in whichfields of the reconstructed frames should be displayed, said apparatuscomprising:means for receiving said compressed video signal; meansresponsive to said received compressed video signal for separating saidinformation DT/DF; means responsive to said received compressed videosignal for decompressing said compressed video signal to provide outputframes of video signal; and means responsive to said information DT/DFfor repeating ones of fields of said output frames and responsive tosaid information DT/DF for providing respective output fields in apredetermined time sequence.
 2. Apparatus for receiving a compressedvideo signal wherein ones of similar fields of video signal have beenexcised and frames of video signal were reconstructed from the remainingfields, said compressed video signal including flag data DT/DFindicative of fields similar to excised fields and which should berepeated, and which field of respective frames should be displayedfirst, said apparatus comprising:means for receiving said compressedvideo signal; means responsive to said received compressed video signalfor separating said information DT/DF; means responsive to said receivedcompressed video signal for decompressing said compressed video signalto provide fields of output video signal; and means responsive to saidinformation DT/DF for repeating ones of said output video fields andresponsive to said information DT/DF for providing respective outputfields in a predetermined sequence.
 3. Apparatus for receiving acompressed video signal including information DT/DF indicative of fieldsof compressed video signal to be displayed more than once afterdecompression and the order of display of decompressed fields, saidapparatus comprising:means for receiving said compressed video signal;means responsive to said received compressed video signal for separatingsaid information DT/DF; means responsive to said received compressedvideo signal for decompressing said compressed video signal to provideoutput frames of video signal; and means responsive to said informationDT/DF for repeating ones of fields of said output frames.
 4. Theapparatus set forth in claim 3 wherein said means responsive to saidinformation DT/DF further includes means responsive to said informationDT/DF for providing respective output fields in a predetermined timesequence.